{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Customer Churn Prediction with XGBoost\n",
    "_**Using Gradient Boosted Trees to Predict Mobile Customer Departure**_\n",
    "\n",
    "---\n",
    "\n",
    "---\n",
    "\n",
    "## Contents\n",
    "\n",
    "1. [Background](#Background)\n",
    "1. [Data](#Data)\n",
    "1. [Train](#Train)\n",
    "1. [Host](#Host)\n",
    "1. [Monitor](#Extensions)\n",
    "\n",
    "---\n",
    "\n",
    "## Background\n",
    "\n",
    "_This notebook has been adapted from an [AWS blog post](https://aws.amazon.com/blogs/ai/predicting-customer-churn-with-amazon-machine-learning/)_\n",
    "\n",
    "Losing customers is costly for any business.  Identifying unhappy customers early on gives you a chance to offer them incentives to stay.  This notebook describes using machine learning (ML) for the automated identification of unhappy customers, also known as customer churn prediction. It uses various features of SageMaker for managing experiments, training the model and monitoring the model after it has been deployed. \n",
    "\n",
    "Let's import the Python libraries we'll need for the remainder of the exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already up-to-date: sagemaker in /usr/local/lib/python3.6/site-packages (1.50.9)\n",
      "Requirement already satisfied, skipping upgrade: packaging>=20.0 in /usr/local/lib/python3.6/site-packages (from sagemaker) (20.1)\n",
      "Requirement already satisfied, skipping upgrade: smdebug-rulesconfig==0.1.2 in /usr/local/lib/python3.6/site-packages (from sagemaker) (0.1.2)\n",
      "Requirement already satisfied, skipping upgrade: scipy>=0.19.0 in /usr/local/lib/python3.6/site-packages (from sagemaker) (1.4.1)\n",
      "Requirement already satisfied, skipping upgrade: requests<3,>=2.20.0 in /usr/local/lib/python3.6/site-packages (from sagemaker) (2.22.0)\n",
      "Requirement already satisfied, skipping upgrade: protobuf>=3.1 in /usr/local/lib/python3.6/site-packages (from sagemaker) (3.11.3)\n",
      "Requirement already satisfied, skipping upgrade: numpy>=1.9.0 in /usr/local/lib/python3.6/site-packages (from sagemaker) (1.18.1)\n",
      "Requirement already satisfied, skipping upgrade: protobuf3-to-dict>=0.1.5 in /usr/local/lib/python3.6/site-packages (from sagemaker) (0.1.5)\n",
      "Requirement already satisfied, skipping upgrade: importlib-metadata>=1.4.0 in /usr/local/lib/python3.6/site-packages (from sagemaker) (1.5.0)\n",
      "Requirement already satisfied, skipping upgrade: boto3>=1.10.44 in /usr/local/lib/python3.6/site-packages (from sagemaker) (1.11.10)\n",
      "Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/site-packages (from packaging>=20.0->sagemaker) (1.14.0)\n",
      "Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /usr/local/lib/python3.6/site-packages (from packaging>=20.0->sagemaker) (2.4.6)\n",
      "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests<3,>=2.20.0->sagemaker) (2019.11.28)\n",
      "Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests<3,>=2.20.0->sagemaker) (1.25.8)\n",
      "Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/site-packages (from requests<3,>=2.20.0->sagemaker) (3.0.4)\n",
      "Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /usr/local/lib/python3.6/site-packages (from requests<3,>=2.20.0->sagemaker) (2.8)\n",
      "Requirement already satisfied, skipping upgrade: setuptools in /usr/local/lib/python3.6/site-packages (from protobuf>=3.1->sagemaker) (44.0.0)\n",
      "Requirement already satisfied, skipping upgrade: zipp>=0.5 in /usr/local/lib/python3.6/site-packages (from importlib-metadata>=1.4.0->sagemaker) (2.1.0)\n",
      "Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/site-packages (from boto3>=1.10.44->sagemaker) (0.9.4)\n",
      "Requirement already satisfied, skipping upgrade: botocore<1.15.0,>=1.14.10 in /usr/local/lib/python3.6/site-packages (from boto3>=1.10.44->sagemaker) (1.14.10)\n",
      "Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/site-packages (from boto3>=1.10.44->sagemaker) (0.3.2)\n",
      "Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/site-packages (from botocore<1.15.0,>=1.14.10->boto3>=1.10.44->sagemaker) (2.8.1)\n",
      "Requirement already satisfied, skipping upgrade: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/site-packages (from botocore<1.15.0,>=1.14.10->boto3>=1.10.44->sagemaker) (0.15.2)\n",
      "\u001b[33mWARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
      "Requirement already satisfied: sagemaker-experiments in /usr/local/lib/python3.6/site-packages (0.1.7)\n",
      "Requirement already satisfied: boto3>=1.10.32 in /usr/local/lib/python3.6/site-packages (from sagemaker-experiments) (1.11.10)\n",
      "Requirement already satisfied: botocore<1.15.0,>=1.14.10 in /usr/local/lib/python3.6/site-packages (from boto3>=1.10.32->sagemaker-experiments) (1.14.10)\n",
      "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/site-packages (from boto3>=1.10.32->sagemaker-experiments) (0.9.4)\n",
      "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/site-packages (from boto3>=1.10.32->sagemaker-experiments) (0.3.2)\n",
      "Requirement already satisfied: urllib3<1.26,>=1.20 in /usr/local/lib/python3.6/site-packages (from botocore<1.15.0,>=1.14.10->boto3>=1.10.32->sagemaker-experiments) (1.25.8)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/site-packages (from botocore<1.15.0,>=1.14.10->boto3>=1.10.32->sagemaker-experiments) (2.8.1)\n",
      "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/site-packages (from botocore<1.15.0,>=1.14.10->boto3>=1.10.32->sagemaker-experiments) (0.15.2)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.15.0,>=1.14.10->boto3>=1.10.32->sagemaker-experiments) (1.14.0)\n",
      "\u001b[33mWARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
      "Requirement already satisfied: pandas in /usr/local/lib/python3.6/site-packages (1.0.0)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/site-packages (from pandas) (2019.3)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/site-packages (from pandas) (1.18.1)\n",
      "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/site-packages (from pandas) (2.8.1)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/site-packages (from python-dateutil>=2.6.1->pandas) (1.14.0)\n",
      "\u001b[33mWARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
      "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/site-packages (3.1.3)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/site-packages (from matplotlib) (1.1.0)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/site-packages (from matplotlib) (2.8.1)\n",
      "Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.6/site-packages (from matplotlib) (1.18.1)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/site-packages (from matplotlib) (2.4.6)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/site-packages (from matplotlib) (0.10.0)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages (from kiwisolver>=1.0.1->matplotlib) (44.0.0)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/site-packages (from python-dateutil>=2.1->matplotlib) (1.14.0)\n",
      "\u001b[33mWARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
      "Collecting smdebug\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/bf/2d/d8a1692b06701b6f00ca21931c5546b6abff21423855d923e45333f71f7c/smdebug-0.5.0.post0-py2.py3-none-any.whl (149kB)\n",
      "\u001b[K     |################################| 153kB 3.1MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.6/site-packages (from smdebug) (3.11.3)\n",
      "Requirement already satisfied: packaging in /usr/local/lib/python3.6/site-packages (from smdebug) (20.1)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.6/site-packages (from smdebug) (1.18.1)\n",
      "Requirement already satisfied: boto3>=1.10.32 in /usr/local/lib/python3.6/site-packages (from smdebug) (1.11.10)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages (from protobuf>=3.6.0->smdebug) (44.0.0)\n",
      "Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.6/site-packages (from protobuf>=3.6.0->smdebug) (1.14.0)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/site-packages (from packaging->smdebug) (2.4.6)\n",
      "Requirement already satisfied: botocore<1.15.0,>=1.14.10 in /usr/local/lib/python3.6/site-packages (from boto3>=1.10.32->smdebug) (1.14.10)\n",
      "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/site-packages (from boto3>=1.10.32->smdebug) (0.3.2)\n",
      "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/site-packages (from boto3>=1.10.32->smdebug) (0.9.4)\n",
      "Requirement already satisfied: urllib3<1.26,>=1.20 in /usr/local/lib/python3.6/site-packages (from botocore<1.15.0,>=1.14.10->boto3>=1.10.32->smdebug) (1.25.8)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/site-packages (from botocore<1.15.0,>=1.14.10->boto3>=1.10.32->smdebug) (2.8.1)\n",
      "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/site-packages (from botocore<1.15.0,>=1.14.10->boto3>=1.10.32->smdebug) (0.15.2)\n",
      "Installing collected packages: smdebug\n",
      "Successfully installed smdebug-0.5.0.post0\n",
      "\u001b[33mWARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install sagemaker -U\n",
    "!{sys.executable} -m pip install sagemaker-experiments\n",
    "!{sys.executable} -m pip install pandas\n",
    "!{sys.executable} -m pip install matplotlib\n",
    "!{sys.executable} -m pip install smdebug"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import io\n",
    "import os\n",
    "import sys\n",
    "import time\n",
    "import json\n",
    "from IPython.display import display\n",
    "from time import strftime, gmtime\n",
    "import boto3\n",
    "import re\n",
    "\n",
    "\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.predictor import csv_serializer\n",
    "from sagemaker.debugger import rule_configs, Rule, DebuggerHookConfig\n",
    "from sagemaker.model_monitor import DataCaptureConfig, DatasetFormat, DefaultModelMonitor\n",
    "from sagemaker.s3 import S3Uploader, S3Downloader\n",
    "\n",
    "from smexperiments.experiment import Experiment\n",
    "from smexperiments.trial import Trial\n",
    "from smexperiments.trial_component import TrialComponent\n",
    "from smexperiments.tracker import Tracker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess = boto3.Session()\n",
    "sm = sess.client('sagemaker')\n",
    "role = sagemaker.get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Data\n",
    "\n",
    "Mobile operators have historical records on which customers ultimately ended up churning and which continued using the service. We can use this historical information to construct train an ML model that can predict customer churn. After training the model, we can pass the profile information of an arbitrary customer (the same profile information that we used to train the model) to the model, and have the model predict whether this customer is going to churn.\n",
    "\n",
    "The dataset we use is publicly available and was mentioned in the book [Discovering Knowledge in Data](https://www.amazon.com/dp/0470908742/) by Daniel T. Larose. It is attributed by the author to the University of California Irvine Repository of Machine Learning Datasets. We already have the dataset downloaded and processed. It's been split into a training set and a validation set. To see how the dataset was preprocessed take a look at this [notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_applying_machine_learning/xgboost_customer_churn/xgboost_customer_churn.ipynb).  \n",
    "\n",
    "Now we'll upload the files to S3 for training but first we will create an S3 bucket for the data if one does not already exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "An error occurred (BucketAlreadyOwnedByYou) when calling the CreateBucket operation: Your previous request to create the named bucket succeeded and you already own it.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'s3://sagemaker-studio-us-east-2-835319576252/xgboost-churn/validation/validation.csv'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "account_id = sess.client('sts', region_name=sess.region_name).get_caller_identity()[\"Account\"]\n",
    "bucket = 'sagemaker-studio-{}-{}'.format(sess.region_name, account_id)\n",
    "prefix = 'xgboost-churn'\n",
    "\n",
    "try:\n",
    "    if sess.region_name == \"us-east-1\":\n",
    "        sess.client('s3').create_bucket(Bucket=bucket)\n",
    "    else:\n",
    "        sess.client('s3').create_bucket(Bucket=bucket, \n",
    "                                        CreateBucketConfiguration={'LocationConstraint': sess.region_name})\n",
    "except Exception as e:\n",
    "    print(e)\n",
    "    \n",
    "S3Uploader.upload('data/train.csv', 's3://{}/{}/{}'.format(bucket, prefix,'train'))\n",
    "S3Uploader.upload('data/validation.csv', 's3://{}/{}/{}'.format(bucket, prefix,'validation'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Train\n",
    "\n",
    "Moving onto training! We will be training a class of models known as gradient boosted decision trees on the data we just uploaded. \n",
    "\n",
    "Because we're using XGBoost, first we'll need to specify the locations of the XGBoost algorithm containers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "docker_image_name = get_image_uri(boto3.Session().region_name, 'xgboost', repo_version='0.90-1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, because we're training with the CSV file format, we'll create `s3_input`s that our training function can use as a pointer to the files in S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_input_train = sagemaker.s3_input(s3_data='s3://{}/{}/train'.format(bucket, prefix), content_type='csv')\n",
    "s3_input_validation = sagemaker.s3_input(s3_data='s3://{}/{}/validation/'.format(bucket, prefix), content_type='csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiment Tracking\n",
    "\n",
    "SageMaker experiment management now allows us to keep track of model training, organize related models together, log model configuration, parameters, and metrics in order to reproduce and iterate on previous models and compare models. We will be creating a single experiment to keep track of the different approaches we will try to train the model.\n",
    "\n",
    "Each approach or training code we run will be an experiment trial and we will be able to compare different trials in SageMaker studio.\n",
    "\n",
    "Let's create the experiment now.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess = sagemaker.session.Session()\n",
    "\n",
    "create_date = strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "customer_churn_experiment = Experiment.create(experiment_name=\"customer-churn-prediction-xgboost-{}\".format(create_date), \n",
    "                                              description=\"Using xgboost to predict customer churn\", \n",
    "                                              sagemaker_boto_client=boto3.client('sagemaker'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameters\n",
    "Now, we can specify our XGBoost hyperparameters.  A few key hyperparameters are:\n",
    "- `max_depth` controls how deep each tree within the algorithm can be built.  Deeper trees can lead to better fit, but are more computationally expensive and can lead to overfitting.  There is typically some trade-off in model performance that needs to be explored between a large number of shallow trees and a smaller number of deeper trees.\n",
    "- `subsample` controls sampling of the training data.  This technique can help reduce overfitting, but setting it too low can also starve the model of data.\n",
    "- `num_round` controls the number of boosting rounds.  This is essentially the subsequent models that are trained using the residuals of previous iterations.  Again, more rounds should produce a better fit on the training data, but can be computationally expensive or lead to overfitting.\n",
    "- `eta` controls how aggressive each round of boosting is.  Larger values lead to more conservative boosting.\n",
    "- `gamma` controls how aggressively trees are grown.  Larger values lead to more conservative models.\n",
    "\n",
    "More detail on XGBoost's hyperparmeters can be found on their GitHub [page](https://github.com/dmlc/xgboost/blob/master/doc/parameter.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparams = {\"max_depth\":5,\n",
    "               \"subsample\":0.8,\n",
    "               \"num_round\":100,\n",
    "               \"eta\":0.2,\n",
    "               \"gamma\":4,\n",
    "               \"min_child_weight\":6,\n",
    "               \"silent\":0,\n",
    "               \"objective\":'binary:logistic'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trial 1 - XGBoost in Algorithm mode\n",
    "\n",
    "For our first trial, we will use the built-in xgboost container to train a model without supplying any additional code. This way, we can use XGBoost to train and deploy a model as you would other built-in Amazon SageMaker algorithms.\n",
    "\n",
    "We will create a new `Trial` object for this and associate the trial with the experiment we created earlier. To train the model we will create an estimator and specify a few parameters like what type of training instances we'd like to use and how many as well as where the trained model artifacts should be stored. \n",
    "\n",
    "We will also associate the training job (when we call `estimator.fit`) with the experiment trial that we just created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating training-job with name: demo-xgboost-customer-churn-2020-02-05-17-38-18-321\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-05 17:38:18 Starting - Starting the training job...\n",
      "2020-02-05 17:38:19 Starting - Launching requested ML instances...\n",
      "2020-02-05 17:39:17 Starting - Preparing the instances for training.........\n",
      "2020-02-05 17:40:23 Downloading - Downloading input data...\n",
      "2020-02-05 17:41:18 Training - Training image download completed. Training in progress.\n",
      "2020-02-05 17:41:18 Uploading - Uploading generated training model.\u001b[34mINFO:sagemaker-containers:Imported framework sagemaker_xgboost_container.training\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:Failed to parse hyperparameter objective value binary:logistic to Json.\u001b[0m\n",
      "\u001b[34mReturning the value itself\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34mINFO:sagemaker_xgboost_container.training:Running XGBoost Sagemaker in algorithm mode\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[17:41:15] 2333x69 matrix with 160977 entries loaded from /opt/ml/input/data/train?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[17:41:15] 666x69 matrix with 45954 entries loaded from /opt/ml/input/data/validation?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[34mINFO:root:Single node training.\u001b[0m\n",
      "\u001b[34mINFO:root:Train matrix has 2333 rows\u001b[0m\n",
      "\u001b[34mINFO:root:Validation matrix has 666 rows\u001b[0m\n",
      "\u001b[34m[0]#011train-error:0.077154#011validation-error:0.099099\u001b[0m\n",
      "\u001b[34m[1]#011train-error:0.050579#011validation-error:0.081081\u001b[0m\n",
      "\u001b[34m[2]#011train-error:0.048864#011validation-error:0.075075\u001b[0m\n",
      "\u001b[34m[3]#011train-error:0.046721#011validation-error:0.072072\u001b[0m\n",
      "\u001b[34m[4]#011train-error:0.048007#011validation-error:0.073574\u001b[0m\n",
      "\u001b[34m[5]#011train-error:0.046721#011validation-error:0.070571\u001b[0m\n",
      "\u001b[34m[6]#011train-error:0.045435#011validation-error:0.073574\u001b[0m\n",
      "\u001b[34m[7]#011train-error:0.043721#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[8]#011train-error:0.045006#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[9]#011train-error:0.042435#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[10]#011train-error:0.040291#011validation-error:0.064565\u001b[0m\n",
      "\u001b[34m[11]#011train-error:0.039006#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[12]#011train-error:0.038577#011validation-error:0.064565\u001b[0m\n",
      "\u001b[34m[13]#011train-error:0.03772#011validation-error:0.064565\u001b[0m\n",
      "\u001b[34m[14]#011train-error:0.03772#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[15]#011train-error:0.039434#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[16]#011train-error:0.038577#011validation-error:0.063063\u001b[0m\n",
      "\u001b[34m[17]#011train-error:0.03772#011validation-error:0.064565\u001b[0m\n",
      "\u001b[34m[18]#011train-error:0.039434#011validation-error:0.063063\u001b[0m\n",
      "\u001b[34m[19]#011train-error:0.039863#011validation-error:0.06006\u001b[0m\n",
      "\u001b[34m[20]#011train-error:0.039434#011validation-error:0.063063\u001b[0m\n",
      "\u001b[34m[21]#011train-error:0.038577#011validation-error:0.063063\u001b[0m\n",
      "\u001b[34m[22]#011train-error:0.038148#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[23]#011train-error:0.036862#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[24]#011train-error:0.036005#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[25]#011train-error:0.034291#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[26]#011train-error:0.033862#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[27]#011train-error:0.033862#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[28]#011train-error:0.033862#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[29]#011train-error:0.033862#011validation-error:0.072072\u001b[0m\n",
      "\u001b[34m[30]#011train-error:0.033862#011validation-error:0.070571\u001b[0m\n",
      "\u001b[34m[31]#011train-error:0.034291#011validation-error:0.072072\u001b[0m\n",
      "\u001b[34m[32]#011train-error:0.034719#011validation-error:0.070571\u001b[0m\n",
      "\u001b[34m[33]#011train-error:0.034719#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[34]#011train-error:0.033005#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[35]#011train-error:0.033862#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[36]#011train-error:0.033862#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[37]#011train-error:0.033005#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[38]#011train-error:0.033433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[39]#011train-error:0.033005#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[40]#011train-error:0.031719#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[41]#011train-error:0.031719#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[42]#011train-error:0.030862#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[43]#011train-error:0.03129#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[44]#011train-error:0.030862#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[45]#011train-error:0.03129#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[46]#011train-error:0.030862#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[47]#011train-error:0.030862#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[48]#011train-error:0.030433#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[49]#011train-error:0.030004#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[50]#011train-error:0.030004#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[51]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[52]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[53]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[54]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[55]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[56]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[57]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[58]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[59]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[60]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[61]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[62]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[63]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[64]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[65]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[66]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[67]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[68]#011train-error:0.029147#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[69]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[70]#011train-error:0.029147#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[71]#011train-error:0.028718#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[72]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[73]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[74]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[75]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[76]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[77]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[78]#011train-error:0.029576#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[79]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[80]#011train-error:0.029576#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[81]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[82]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[83]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[84]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[85]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[86]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[87]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[88]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[89]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[90]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[91]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[92]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[93]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[94]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[95]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[96]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[97]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[98]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[99]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\n",
      "2020-02-05 17:41:25 Completed - Training job completed\n",
      "Training seconds: 62\n",
      "Billable seconds: 62\n"
     ]
    }
   ],
   "source": [
    "trial = Trial.create(trial_name=\"algorithm-mode-trial-{}\".format(strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())), \n",
    "                     experiment_name=customer_churn_experiment.experiment_name,\n",
    "                     sagemaker_boto_client=boto3.client('sagemaker'))\n",
    "\n",
    "xgb = sagemaker.estimator.Estimator(image_name=docker_image_name,\n",
    "                                    role=role,\n",
    "                                    hyperparameters=hyperparams,\n",
    "                                    train_instance_count=1, \n",
    "                                    train_instance_type='ml.m4.xlarge',\n",
    "                                    output_path='s3://{}/{}/output'.format(bucket, prefix),\n",
    "                                    base_job_name=\"demo-xgboost-customer-churn\",\n",
    "                                    sagemaker_session=sess)\n",
    "\n",
    "xgb.fit({'train': s3_input_train,\n",
    "         'validation': s3_input_validation}, \n",
    "        experiment_config={\n",
    "            \"ExperimentName\": customer_churn_experiment.experiment_name, \n",
    "            \"TrialName\": trial.trial_name,\n",
    "            \"TrialComponentDisplayName\": \"Training\",\n",
    "        }\n",
    "       )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the training job kicks off and succeeds, you should be able to view metrics, logs and graphs related to the trial in experiments tab in SageMaker Studio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trial 2 - XGBoost in Framework mode\n",
    "\n",
    "To get even more flexibility we will try to train a similar model but using XGBoost in framework mode. This way of using XGBoost should be to familiar to users who have worked with the open source XGBoost. Using XGBoost as a framework provides more flexibility than using it as a built-in algorithm as it enables more advanced scenarios that allow pre-processing and post-processing scripts to be incorporated into your training script. Specifically, we will be able to specify a list of rules that we want the SageMaker Debugger to evaluate our training process against."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specify SageMaker Debug Rules\n",
    "\n",
    "Amazon SageMaker now enables debugging of machine learning models during training. During training the debugger periodicially saves tensors, which fully specify the state of the machine learning model at that instance. These tensors are saved to S3 for analysis and visualization to diagnose training issues using SageMaker studio.\n",
    "\n",
    "In order to enable automated detection of common issues during machine learning training, the SageMaker debugger also allows you to attach a list of rules to evaluate the training job against.\n",
    "\n",
    "Some rule configs that apply to XGBoost include `AllZero`, `ClassImbalance`, `Confusion`, `LossNotDecreasing`, `Overfit`, `Overtraining`, `SimilarAcrossRuns`, `TensorVariance`, `UnchangedTensor`, `TreeDepth`. \n",
    "\n",
    "Here we will use the just the `LossNotDecreasing` rule which is triggered if the loss at does not decrease monotonically at any point during training. Let's create the rule now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "debug_rules = [Rule.sagemaker(rule_configs.loss_not_decreasing())]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fit Estimator\n",
    "\n",
    "In order to use XGBoost as a framework you need to specify an entry-point script that can incorporate additional processing into your training jobs.\n",
    "\n",
    "We have made a couple of simple changes to the  enable the SageMaker Debugger `smdebug`. Here we created a SessionHook which we pass as a callback function when creating a Booster. We passed a SaveConfig object telling the hook to save the evaluation metrics, feature importances, and SHAP values at regular intervals. Note that Sagemaker-Debugger is highly configurable, you can choose exactly what to save. The changes are described in a bit more detail below after we train this example as well as in even more detail in our [Developer Guide for XGBoost](https://github.com/awslabs/sagemaker-debugger/tree/master/docs/xgboost)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error: cannot read infile: [Errno 2] No such file or directory: 'xgboost_customer_churn.py'\n"
     ]
    }
   ],
   "source": [
    "!pygmentize xgboost_customer_churn.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create our Framwork estimator and call `fit` to start the training job. As before, we will create a separate trial for this run so that we can compare later using SageMaker Studio. Since we are running in framework mode we also need to pass additional parameters like the entry point script, and the framework version to the estimator. \n",
    "\n",
    "As training progresses, you will be able to see logs from the SageMaker debugger evaluating the rule against the training job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating training-job with name: demo-xgboost-customer-churn-2020-02-05-17-56-22-154\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-05 17:56:22 Starting - Starting the training job...\n",
      "2020-02-05 17:56:45 Starting - Launching requested ML instances\n",
      "********* Debugger Rule Status *********\n",
      "*\n",
      "*  LossNotDecreasing: InProgress        \n",
      "*\n",
      "****************************************\n",
      "......\n",
      "2020-02-05 17:57:46 Starting - Preparing the instances for training......\n",
      "2020-02-05 17:58:47 Downloading - Downloading input data...\n",
      "2020-02-05 17:59:07 Training - Downloading the training image..\u001b[34mINFO:sagemaker-containers:Imported framework sagemaker_xgboost_container.training\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34mINFO:sagemaker_xgboost_container.training:Invoking user training script.\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:Module xgboost_customer_churn does not provide a setup.py. \u001b[0m\n",
      "\u001b[34mGenerating setup.py\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:Generating setup.cfg\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:Generating MANIFEST.in\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:Installing module with the following command:\u001b[0m\n",
      "\u001b[34m/miniconda3/bin/python -m pip install . -r requirements.txt\u001b[0m\n",
      "\u001b[34mProcessing /opt/ml/code\u001b[0m\n",
      "\u001b[34mCollecting smdebug\n",
      "  Downloading https://files.pythonhosted.org/packages/bf/2d/d8a1692b06701b6f00ca21931c5546b6abff21423855d923e45333f71f7c/smdebug-0.5.0.post0-py2.py3-none-any.whl (149kB)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: boto3>=1.10.32 in /miniconda3/lib/python3.7/site-packages (from smdebug->-r requirements.txt (line 1)) (1.10.47)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: protobuf>=3.6.0 in /miniconda3/lib/python3.7/site-packages (from smdebug->-r requirements.txt (line 1)) (3.11.2)\u001b[0m\n",
      "\u001b[34mCollecting packaging\n",
      "  Downloading https://files.pythonhosted.org/packages/98/42/87c585dd3b113c775e65fd6b8d9d0a43abe1819c471d7af702d4e01e9b20/packaging-20.1-py2.py3-none-any.whl\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: numpy in /miniconda3/lib/python3.7/site-packages (from smdebug->-r requirements.txt (line 1)) (1.18.1)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: jmespath<1.0.0,>=0.7.1 in /miniconda3/lib/python3.7/site-packages (from boto3>=1.10.32->smdebug->-r requirements.txt (line 1)) (0.9.4)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: botocore<1.14.0,>=1.13.47 in /miniconda3/lib/python3.7/site-packages (from boto3>=1.10.32->smdebug->-r requirements.txt (line 1)) (1.13.47)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /miniconda3/lib/python3.7/site-packages (from boto3>=1.10.32->smdebug->-r requirements.txt (line 1)) (0.2.1)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: setuptools in /miniconda3/lib/python3.7/site-packages (from protobuf>=3.6.0->smdebug->-r requirements.txt (line 1)) (44.0.0.post20200106)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: six>=1.9 in /miniconda3/lib/python3.7/site-packages (from protobuf>=3.6.0->smdebug->-r requirements.txt (line 1)) (1.13.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: pyparsing>=2.0.2 in /miniconda3/lib/python3.7/site-packages (from packaging->smdebug->-r requirements.txt (line 1)) (2.4.6)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: docutils<0.16,>=0.10 in /miniconda3/lib/python3.7/site-packages (from botocore<1.14.0,>=1.13.47->boto3>=1.10.32->smdebug->-r requirements.txt (line 1)) (0.15.2)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= \"2.7\" in /miniconda3/lib/python3.7/site-packages (from botocore<1.14.0,>=1.13.47->boto3>=1.10.32->smdebug->-r requirements.txt (line 1)) (2.8.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: urllib3<1.26,>=1.20; python_version >= \"3.4\" in /miniconda3/lib/python3.7/site-packages (from botocore<1.14.0,>=1.13.47->boto3>=1.10.32->smdebug->-r requirements.txt (line 1)) (1.24.3)\u001b[0m\n",
      "\u001b[34mBuilding wheels for collected packages: xgboost-customer-churn\n",
      "  Building wheel for xgboost-customer-churn (setup.py): started\n",
      "  Building wheel for xgboost-customer-churn (setup.py): finished with status 'done'\n",
      "  Created wheel for xgboost-customer-churn: filename=xgboost_customer_churn-1.0.0-py2.py3-none-any.whl size=38506 sha256=4fbab488fe73722db23b2ca2787c00fb9dae1dd4368efe7fedc08918f3f0c74d\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-c_i41dr8/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3\u001b[0m\n",
      "\u001b[34mSuccessfully built xgboost-customer-churn\u001b[0m\n",
      "\u001b[34mInstalling collected packages: packaging, smdebug, xgboost-customer-churn\u001b[0m\n",
      "\u001b[34mSuccessfully installed packaging-20.1 smdebug-0.5.0.post0 xgboost-customer-churn-1.0.0\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:Invoking user script\n",
      "\u001b[0m\n",
      "\u001b[34mTraining Env:\n",
      "\u001b[0m\n",
      "\u001b[34m{\n",
      "    \"additional_framework_parameters\": {},\n",
      "    \"channel_input_dirs\": {\n",
      "        \"validation\": \"/opt/ml/input/data/validation\",\n",
      "        \"train\": \"/opt/ml/input/data/train\"\n",
      "    },\n",
      "    \"current_host\": \"algo-1\",\n",
      "    \"framework_module\": \"sagemaker_xgboost_container.training:main\",\n",
      "    \"hosts\": [\n",
      "        \"algo-1\"\n",
      "    ],\n",
      "    \"hyperparameters\": {\n",
      "        \"silent\": 0,\n",
      "        \"max_depth\": 5,\n",
      "        \"objective\": \"binary:logistic\",\n",
      "        \"eta\": 0.2,\n",
      "        \"num_round\": 100,\n",
      "        \"subsample\": 0.8,\n",
      "        \"gamma\": 4,\n",
      "        \"min_child_weight\": 6\n",
      "    },\n",
      "    \"input_config_dir\": \"/opt/ml/input/config\",\n",
      "    \"input_data_config\": {\n",
      "        \"validation\": {\n",
      "            \"ContentType\": \"csv\",\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        },\n",
      "        \"train\": {\n",
      "            \"ContentType\": \"csv\",\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        }\n",
      "    },\n",
      "    \"input_dir\": \"/opt/ml/input\",\n",
      "    \"is_master\": true,\n",
      "    \"job_name\": \"demo-xgboost-customer-churn-2020-02-05-17-56-22-154\",\n",
      "    \"log_level\": 20,\n",
      "    \"master_hostname\": \"algo-1\",\n",
      "    \"model_dir\": \"/opt/ml/model\",\n",
      "    \"module_dir\": \"s3://sagemaker-studio-us-east-2-835319576252/demo-xgboost-customer-churn-2020-02-05-17-56-22-154/source/sourcedir.tar.gz\",\n",
      "    \"module_name\": \"xgboost_customer_churn\",\n",
      "    \"network_interface_name\": \"eth0\",\n",
      "    \"num_cpus\": 4,\n",
      "    \"num_gpus\": 0,\n",
      "    \"output_data_dir\": \"/opt/ml/output/data\",\n",
      "    \"output_dir\": \"/opt/ml/output\",\n",
      "    \"output_intermediate_dir\": \"/opt/ml/output/intermediate\",\n",
      "    \"resource_config\": {\n",
      "        \"current_host\": \"algo-1\",\n",
      "        \"hosts\": [\n",
      "            \"algo-1\"\n",
      "        ],\n",
      "        \"network_interface_name\": \"eth0\"\n",
      "    },\n",
      "    \"user_entry_point\": \"xgboost_customer_churn.py\"\u001b[0m\n",
      "\u001b[34m}\n",
      "\u001b[0m\n",
      "\u001b[34mEnvironment variables:\n",
      "\u001b[0m\n",
      "\u001b[34mSM_HOSTS=[\"algo-1\"]\u001b[0m\n",
      "\u001b[34mSM_NETWORK_INTERFACE_NAME=eth0\u001b[0m\n",
      "\u001b[34mSM_HPS={\"eta\":0.2,\"gamma\":4,\"max_depth\":5,\"min_child_weight\":6,\"num_round\":100,\"objective\":\"binary:logistic\",\"silent\":0,\"subsample\":0.8}\u001b[0m\n",
      "\u001b[34mSM_USER_ENTRY_POINT=xgboost_customer_churn.py\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_PARAMS={}\u001b[0m\n",
      "\u001b[34mSM_RESOURCE_CONFIG={\"current_host\":\"algo-1\",\"hosts\":[\"algo-1\"],\"network_interface_name\":\"eth0\"}\u001b[0m\n",
      "\u001b[34mSM_INPUT_DATA_CONFIG={\"train\":{\"ContentType\":\"csv\",\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"validation\":{\"ContentType\":\"csv\",\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}}\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DATA_DIR=/opt/ml/output/data\u001b[0m\n",
      "\u001b[34mSM_CHANNELS=[\"train\",\"validation\"]\u001b[0m\n",
      "\u001b[34mSM_CURRENT_HOST=algo-1\u001b[0m\n",
      "\u001b[34mSM_MODULE_NAME=xgboost_customer_churn\u001b[0m\n",
      "\u001b[34mSM_LOG_LEVEL=20\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_MODULE=sagemaker_xgboost_container.training:main\u001b[0m\n",
      "\u001b[34mSM_INPUT_DIR=/opt/ml/input\u001b[0m\n",
      "\u001b[34mSM_INPUT_CONFIG_DIR=/opt/ml/input/config\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DIR=/opt/ml/output\u001b[0m\n",
      "\u001b[34mSM_NUM_CPUS=4\u001b[0m\n",
      "\u001b[34mSM_NUM_GPUS=0\u001b[0m\n",
      "\u001b[34mSM_MODEL_DIR=/opt/ml/model\u001b[0m\n",
      "\u001b[34mSM_MODULE_DIR=s3://sagemaker-studio-us-east-2-835319576252/demo-xgboost-customer-churn-2020-02-05-17-56-22-154/source/sourcedir.tar.gz\u001b[0m\n",
      "\u001b[34mSM_TRAINING_ENV={\"additional_framework_parameters\":{},\"channel_input_dirs\":{\"train\":\"/opt/ml/input/data/train\",\"validation\":\"/opt/ml/input/data/validation\"},\"current_host\":\"algo-1\",\"framework_module\":\"sagemaker_xgboost_container.training:main\",\"hosts\":[\"algo-1\"],\"hyperparameters\":{\"eta\":0.2,\"gamma\":4,\"max_depth\":5,\"min_child_weight\":6,\"num_round\":100,\"objective\":\"binary:logistic\",\"silent\":0,\"subsample\":0.8},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{\"train\":{\"ContentType\":\"csv\",\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"validation\":{\"ContentType\":\"csv\",\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}},\"input_dir\":\"/opt/ml/input\",\"is_master\":true,\"job_name\":\"demo-xgboost-customer-churn-2020-02-05-17-56-22-154\",\"log_level\":20,\"master_hostname\":\"algo-1\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://sagemaker-studio-us-east-2-835319576252/demo-xgboost-customer-churn-2020-02-05-17-56-22-154/source/sourcedir.tar.gz\",\"module_name\":\"xgboost_customer_churn\",\"network_interface_name\":\"eth0\",\"num_cpus\":4,\"num_gpus\":0,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_host\":\"algo-1\",\"hosts\":[\"algo-1\"],\"network_interface_name\":\"eth0\"},\"user_entry_point\":\"xgboost_customer_churn.py\"}\u001b[0m\n",
      "\u001b[34mSM_USER_ARGS=[\"--eta\",\"0.2\",\"--gamma\",\"4\",\"--max_depth\",\"5\",\"--min_child_weight\",\"6\",\"--num_round\",\"100\",\"--objective\",\"binary:logistic\",\"--silent\",\"0\",\"--subsample\",\"0.8\"]\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate\u001b[0m\n",
      "\u001b[34mSM_CHANNEL_VALIDATION=/opt/ml/input/data/validation\u001b[0m\n",
      "\u001b[34mSM_CHANNEL_TRAIN=/opt/ml/input/data/train\u001b[0m\n",
      "\u001b[34mSM_HP_SILENT=0\u001b[0m\n",
      "\u001b[34mSM_HP_MAX_DEPTH=5\u001b[0m\n",
      "\u001b[34mSM_HP_OBJECTIVE=binary:logistic\u001b[0m\n",
      "\u001b[34mSM_HP_ETA=0.2\u001b[0m\n",
      "\u001b[34mSM_HP_NUM_ROUND=100\u001b[0m\n",
      "\u001b[34mSM_HP_SUBSAMPLE=0.8\u001b[0m\n",
      "\u001b[34mSM_HP_GAMMA=4\u001b[0m\n",
      "\u001b[34mSM_HP_MIN_CHILD_WEIGHT=6\u001b[0m\n",
      "\u001b[34mPYTHONPATH=/miniconda3/bin:/:/usr/local/lib/python3.5/dist-packages/xgboost/dmlc-core/tracker:/miniconda3/lib/python37.zip:/miniconda3/lib/python3.7:/miniconda3/lib/python3.7/lib-dynload:/miniconda3/lib/python3.7/site-packages\n",
      "\u001b[0m\n",
      "\u001b[34mInvoking script with the following command:\n",
      "\u001b[0m\n",
      "\u001b[34m/miniconda3/bin/python -m xgboost_customer_churn --eta 0.2 --gamma 4 --max_depth 5 --min_child_weight 6 --num_round 100 --objective binary:logistic --silent 0 --subsample 0.8\n",
      "\n",
      "\u001b[0m\n",
      "\u001b[34m[17:59:34] 2333x69 matrix with 160977 entries loaded from /opt/ml/input/data/train?format=csv&label_column=0\u001b[0m\n",
      "\u001b[34m[17:59:34] 666x69 matrix with 45954 entries loaded from /opt/ml/input/data/validation?format=csv&label_column=0\u001b[0m\n",
      "\u001b[34m[2020-02-05 17:59:34.912 ip-10-0-130-113.us-east-2.compute.internal:34 INFO hook.py:152] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[2020-02-05 17:59:34.912 ip-10-0-130-113.us-east-2.compute.internal:34 INFO hook.py:197] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[2020-02-05 17:59:34.930 ip-10-0-130-113.us-east-2.compute.internal:34 INFO hook.py:326] Monitoring the collections: metrics\u001b[0m\n",
      "\u001b[34m[0]#011train-error:0.077154#011validation-error:0.099099\u001b[0m\n",
      "\u001b[34m[1]#011train-error:0.050579#011validation-error:0.081081\u001b[0m\n",
      "\u001b[34m[2]#011train-error:0.048864#011validation-error:0.075075\u001b[0m\n",
      "\u001b[34m[3]#011train-error:0.046721#011validation-error:0.072072\u001b[0m\n",
      "\u001b[34m[4]#011train-error:0.048007#011validation-error:0.073574\u001b[0m\n",
      "\u001b[34m[5]#011train-error:0.046721#011validation-error:0.070571\u001b[0m\n",
      "\u001b[34m[6]#011train-error:0.045435#011validation-error:0.073574\u001b[0m\n",
      "\u001b[34m[7]#011train-error:0.043721#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[8]#011train-error:0.045006#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[9]#011train-error:0.042435#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[10]#011train-error:0.040291#011validation-error:0.064565\u001b[0m\n",
      "\u001b[34m[11]#011train-error:0.039006#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[12]#011train-error:0.038577#011validation-error:0.064565\u001b[0m\n",
      "\u001b[34m[13]#011train-error:0.03772#011validation-error:0.064565\u001b[0m\n",
      "\u001b[34m[14]#011train-error:0.03772#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[15]#011train-error:0.039434#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[16]#011train-error:0.038577#011validation-error:0.063063\u001b[0m\n",
      "\u001b[34m[17]#011train-error:0.03772#011validation-error:0.064565\u001b[0m\n",
      "\u001b[34m[18]#011train-error:0.039434#011validation-error:0.063063\u001b[0m\n",
      "\u001b[34m[19]#011train-error:0.039863#011validation-error:0.06006\u001b[0m\n",
      "\u001b[34m[20]#011train-error:0.039434#011validation-error:0.063063\u001b[0m\n",
      "\u001b[34m[21]#011train-error:0.038577#011validation-error:0.063063\u001b[0m\n",
      "\u001b[34m[22]#011train-error:0.038148#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[23]#011train-error:0.036862#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[24]#011train-error:0.036005#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[25]#011train-error:0.034291#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[26]#011train-error:0.033862#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[27]#011train-error:0.033862#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[28]#011train-error:0.033862#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[29]#011train-error:0.033862#011validation-error:0.072072\u001b[0m\n",
      "\u001b[34m[30]#011train-error:0.033862#011validation-error:0.070571\u001b[0m\n",
      "\u001b[34m[31]#011train-error:0.034291#011validation-error:0.072072\u001b[0m\n",
      "\u001b[34m[32]#011train-error:0.034719#011validation-error:0.070571\u001b[0m\n",
      "\u001b[34m[33]#011train-error:0.034719#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[34]#011train-error:0.033005#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[35]#011train-error:0.033862#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[36]#011train-error:0.033862#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[37]#011train-error:0.033005#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[38]#011train-error:0.033433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[39]#011train-error:0.033005#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[40]#011train-error:0.031719#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[41]#011train-error:0.031719#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[42]#011train-error:0.030862#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[43]#011train-error:0.03129#011validation-error:0.069069\u001b[0m\n",
      "\u001b[34m[44]#011train-error:0.030862#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[45]#011train-error:0.03129#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[46]#011train-error:0.030862#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[47]#011train-error:0.030862#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[48]#011train-error:0.030433#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[49]#011train-error:0.030004#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[50]#011train-error:0.030004#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[51]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[52]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[53]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[54]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[55]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[56]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[57]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[58]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[59]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[60]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[61]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[62]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[63]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[64]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[65]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[66]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[67]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[68]#011train-error:0.029147#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[69]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[70]#011train-error:0.029147#011validation-error:0.067568\u001b[0m\n",
      "\u001b[34m[71]#011train-error:0.028718#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[72]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[73]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[74]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[75]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[76]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[77]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[78]#011train-error:0.029576#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[79]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[80]#011train-error:0.029576#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[81]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[82]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[83]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[84]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[85]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[86]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[87]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[88]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[89]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[90]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[91]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[92]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[93]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[94]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[95]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[96]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[97]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[98]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[99]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n",
      "\u001b[34m[2020-02-05 17:59:36.940 ip-10-0-130-113.us-east-2.compute.internal:34 INFO utils.py:25] The end of training job file will not be written for jobs running under SageMaker.\u001b[0m\n",
      "\n",
      "2020-02-05 17:59:47 Uploading - Uploading generated training model\n",
      "2020-02-05 17:59:47 Completed - Training job completed\n",
      "Training seconds: 67\n",
      "Billable seconds: 67\n"
     ]
    }
   ],
   "source": [
    "entry_point_script = \"xgboost_customer_churn.py\"\n",
    "\n",
    "trial = Trial.create(trial_name=\"framework-mode-trial-{}\".format(strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())), \n",
    "                     experiment_name=customer_churn_experiment.experiment_name,\n",
    "                     sagemaker_boto_client=boto3.client('sagemaker'))\n",
    "\n",
    "framework_xgb = sagemaker.xgboost.XGBoost(image_name=docker_image_name,\n",
    "                                          entry_point=entry_point_script,\n",
    "                                          source_dir='./amazon-sagemaker-studio/churn/',\n",
    "                                          role=role,\n",
    "                                          framework_version=\"0.90-1\",\n",
    "                                          py_version=\"py3\",\n",
    "                                          hyperparameters=hyperparams,\n",
    "                                          train_instance_count=1, \n",
    "                                          train_instance_type='ml.m4.xlarge',\n",
    "                                          output_path='s3://{}/{}/output'.format(bucket, prefix),\n",
    "                                          base_job_name=\"demo-xgboost-customer-churn\",\n",
    "                                          sagemaker_session=sess,\n",
    "                                          rules=debug_rules\n",
    "                                          )\n",
    "\n",
    "framework_xgb.fit({'train': s3_input_train,\n",
    "                   'validation': s3_input_validation}, \n",
    "                  experiment_config={\n",
    "                      \"ExperimentName\": customer_churn_experiment.experiment_name, \n",
    "                      \"TrialName\": trial.trial_name,\n",
    "                      \"TrialComponentDisplayName\": \"Training\",\n",
    "                  })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Host\n",
    "\n",
    "Now that we've trained the model, let's deploy it to a hosted endpoint. In other to monitor the model once it is hosted and serving requests, we will also add configurations to capture data being sent to the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EndpointName=demo-xgboost-customer-churn-2020-02-05-18-00-04\n"
     ]
    }
   ],
   "source": [
    "data_capture_prefix = '{}/datacapture'.format(prefix)\n",
    "\n",
    "endpoint_name = \"demo-xgboost-customer-churn-\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "print(\"EndpointName={}\".format(endpoint_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model with name: demo-xgboost-customer-churn-2020-02-05-17-38-18-321\n",
      "INFO:sagemaker:Creating endpoint with name demo-xgboost-customer-churn-2020-02-05-18-00-04\n"
     ]
    },
    {
     "ename": "ResourceLimitExceeded",
     "evalue": "An error occurred (ResourceLimitExceeded) when calling the CreateEndpoint operation: The account-level service limit 'ml.m4.xlarge for endpoint usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances. Please contact AWS support to request an increase for this limit.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mResourceLimitExceeded\u001b[0m                     Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-26-03a172b80fc0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      4\u001b[0m                            data_capture_config=DataCaptureConfig(enable_capture=True,\n\u001b[1;32m      5\u001b[0m                                                                  \u001b[0msampling_percentage\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m                                                                  \u001b[0mdestination_s3_uri\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m's3://{}/{}'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbucket\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata_capture_prefix\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m                                                                 )\n\u001b[1;32m      8\u001b[0m                            )\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/sagemaker/estimator.py\u001b[0m in \u001b[0;36mdeploy\u001b[0;34m(self, initial_instance_count, instance_type, accelerator_type, endpoint_name, use_compiled_model, update_endpoint, wait, model_name, kms_key, data_capture_config, tags, **kwargs)\u001b[0m\n\u001b[1;32m    691\u001b[0m             \u001b[0mwait\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    692\u001b[0m             \u001b[0mkms_key\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkms_key\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 693\u001b[0;31m             \u001b[0mdata_capture_config\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdata_capture_config\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    694\u001b[0m         )\n\u001b[1;32m    695\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/sagemaker/model.py\u001b[0m in \u001b[0;36mdeploy\u001b[0;34m(self, initial_instance_count, instance_type, accelerator_type, endpoint_name, update_endpoint, tags, kms_key, wait, data_capture_config)\u001b[0m\n\u001b[1;32m    484\u001b[0m                 \u001b[0mkms_key\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkms_key\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    485\u001b[0m                 \u001b[0mwait\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 486\u001b[0;31m                 \u001b[0mdata_capture_config_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdata_capture_config_dict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    487\u001b[0m             )\n\u001b[1;32m    488\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36mendpoint_from_production_variants\u001b[0;34m(self, name, production_variants, tags, kms_key, wait, data_capture_config_dict)\u001b[0m\n\u001b[1;32m   2847\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2848\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msagemaker_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate_endpoint_config\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mconfig_options\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2849\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate_endpoint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mendpoint_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconfig_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtags\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtags\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwait\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2850\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2851\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mexpand_role\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrole\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36mcreate_endpoint\u001b[0;34m(self, endpoint_name, config_name, tags, wait)\u001b[0m\n\u001b[1;32m   2376\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2377\u001b[0m         self.sagemaker_client.create_endpoint(\n\u001b[0;32m-> 2378\u001b[0;31m             \u001b[0mEndpointName\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mendpoint_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mEndpointConfigName\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mconfig_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTags\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtags\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2379\u001b[0m         )\n\u001b[1;32m   2380\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mwait\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_api_call\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    274\u001b[0m                     \"%s() only accepts keyword arguments.\" % py_operation_name)\n\u001b[1;32m    275\u001b[0m             \u001b[0;31m# The \"self\" in this scope is referring to the BaseClient.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 276\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_api_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moperation_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    277\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    278\u001b[0m         \u001b[0m_api_call\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpy_operation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_make_api_call\u001b[0;34m(self, operation_name, api_params)\u001b[0m\n\u001b[1;32m    584\u001b[0m             \u001b[0merror_code\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Error\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Code\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    585\u001b[0m             \u001b[0merror_class\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_code\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merror_code\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 586\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0merror_class\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparsed_response\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moperation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    587\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    588\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mResourceLimitExceeded\u001b[0m: An error occurred (ResourceLimitExceeded) when calling the CreateEndpoint operation: The account-level service limit 'ml.m4.xlarge for endpoint usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances. Please contact AWS support to request an increase for this limit."
     ]
    }
   ],
   "source": [
    "xgb_predictor = xgb.deploy(initial_instance_count=1, \n",
    "                           instance_type='ml.m4.xlarge',\n",
    "                           endpoint_name=endpoint_name,\n",
    "                           data_capture_config=DataCaptureConfig(enable_capture=True,\n",
    "                                                                 sampling_percentage=100,\n",
    "                                                                 destination_s3_uri='s3://{}/{}'.format(bucket, data_capture_prefix)\n",
    "                                                                )\n",
    "                           )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Invoke the deployed model\n",
    "\n",
    "Now that we have a hosted endpoint running, we can make real-time predictions from our model very easily, simply by making an http POST request.  But first, we'll need to setup serializers and deserializers for passing our `test_data` NumPy arrays to the model behind the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_predictor.content_type = 'text/csv'\n",
    "xgb_predictor.serializer = csv_serializer\n",
    "xgb_predictor.deserializer = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll loop over our test dataset and collect predictions by invoking the XGBoost endpoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Sending test traffic to the endpoint {}. \\nPlease wait for a minute...\".format(endpoint_name))\n",
    "\n",
    "with open('data/test_sample.csv', 'r') as f:\n",
    "    for row in f:\n",
    "        payload = row.rstrip('\\n')\n",
    "        response = xgb_predictor.predict(data=payload)\n",
    "        time.sleep(0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify data capture in S3\n",
    "\n",
    "Since we made some real-time predictions by sending data to our endpoint we should have also captured that data for monitoring purposes. \n",
    "\n",
    "Let's list the data capture files stored in S3. You should expect to see different files from different time periods organized based on the hour in which the invocation occurred. The format of the s3 path is:\n",
    "\n",
    "`s3://{destination-bucket-prefix}/{endpoint-name}/{variant-name}/yyyy/mm/dd/hh/filename.jsonl`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "current_endpoint_capture_prefix = '{}/{}'.format(data_capture_prefix, endpoint_name)\n",
    "print(\"Found Data Capture Files:\")\n",
    "capture_files = S3Downloader.list(\"s3://{}/{}\".format(bucket, current_endpoint_capture_prefix))\n",
    "print(capture_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the data captured is stored in a SageMaker specific json-line formatted file. Next, Let's take a quick peek at the contents of a single line in a pretty formatted json so that we can observe the format a little better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "capture_file = S3Downloader.read_file(capture_files[-1])\n",
    "\n",
    "print(\"=====Single Data Capture====\")\n",
    "print(json.dumps(json.loads(capture_file.split('\\n')[0]), indent=2)[:2000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, each inference request is captured in one line in the jsonl file. The line contains both the input and output merged together. In our example, we provided the ContentType as `text/csv` which is reflected in the `observedContentType` value. Also, we expose the enconding that we used to encode the input and output payloads in the capture format with the `encoding` value.\n",
    "\n",
    "To recap, we have observed how you can enable capturing the input and/or output payloads to an Endpoint with a new parameter. We have also observed how the captured format looks like in S3. Let's continue to explore how SageMaker helps with monitoring the data collected in S3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Model Monitoring - Baselining and continous monitoring\n",
    "In addition to collecting the data, SageMaker provides capability for you to monitor and evaluate the data observed by the Endpoints. For this :\n",
    "1. We need to create a baseline with which we compare the realtime traffic against. \n",
    "1. Once a baseline is ready, we can set up a schedule to continously evaluate/compare against the baseline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Constraint suggestion with baseline/training dataset\n",
    "\n",
    "The training dataset with which you trained the model is usually a good baseline dataset. Note that the training dataset data schema and the inference dataset schema should exactly match (ie number and type of the features).\n",
    "\n",
    "From our training dataset let's ask SageMaker to suggest a set of baseline `constraints` and generate descriptive `statistics` to explore the data. For this example, let's upload the training dataset which was used to train model. We will use the dataset file with column headers to have descriptive feature names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_prefix = prefix + '/baselining'\n",
    "baseline_data_prefix = baseline_prefix + '/data'\n",
    "baseline_results_prefix = baseline_prefix + '/results'\n",
    "\n",
    "baseline_data_uri = 's3://{}/{}'.format(bucket,baseline_data_prefix)\n",
    "baseline_results_uri = 's3://{}/{}'.format(bucket, baseline_results_prefix)\n",
    "print('Baseline data uri: {}'.format(baseline_data_uri))\n",
    "print('Baseline results uri: {}'.format(baseline_results_uri))\n",
    "baseline_data_path = S3Uploader.upload(\"data/training-dataset-with-header.csv\", baseline_data_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create a baselining job with training dataset\n",
    "\n",
    "Now that we have the training data ready in S3, let's kick off a job to `suggest` constraints. The convenient helper kicks off a `ProcessingJob` using a SageMaker provided ProcessingJob container to generate the constraints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_default_monitor = DefaultModelMonitor(role=role,\n",
    "                                         instance_count=1,\n",
    "                                         instance_type='ml.m5.xlarge',\n",
    "                                         volume_size_in_gb=20,\n",
    "                                         max_runtime_in_seconds=3600,\n",
    "                                        )\n",
    "\n",
    "baseline_job = my_default_monitor.suggest_baseline(baseline_dataset=baseline_data_path,\n",
    "                                                   dataset_format=DatasetFormat.csv(header=True),\n",
    "                                                   output_s3_uri=baseline_results_uri,\n",
    "                                                   wait=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the job succeeds, we can explore the `baseline_results_uri` location in s3 to see what files where stored there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Found Files:\")\n",
    "S3Downloader.list(\"s3://{}/{}\".format(bucket, baseline_results_prefix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a`constraints.json` file that has information about suggested constraints. We also have a `statistics.json` which contains statistical information about the data in the baseline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_job = my_default_monitor.latest_baselining_job\n",
    "schema_df = pd.io.json.json_normalize(baseline_job.baseline_statistics().body_dict[\"features\"])\n",
    "schema_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "constraints_df = pd.io.json.json_normalize(baseline_job.suggested_constraints().body_dict[\"features\"])\n",
    "constraints_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Analyzing Subsequent captures for data quality issues\n",
    "\n",
    "Now, that we have generated a baseline dataset and processed the baseline dataset to get baseline statistics and constraints, let's proceed monitor and analyze the data being sent to the endpoitn with Monitoring Schedules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create a schedule\n",
    "Let's first create a Monitoring schedule for the previously created Endpoint. The schedule specifies the cadence at which we run a new processing job to compare recent data captures to the baseline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, copy over some test scripts to the S3 bucket so that they can be used for pre and post processing\n",
    "code_prefix = '{}/code'.format(prefix)\n",
    "S3Uploader.upload('preprocessor.py', 's3://{}/{}'.format(bucket,code_prefix))\n",
    "S3Uploader.upload('postprocessor.py', 's3://{}/{}'.format(bucket,code_prefix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are ready to create a model monitoring schedule for the Endpoint created before and also the baseline resources (constraints and statistics) which were generated above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.model_monitor import CronExpressionGenerator\n",
    "from time import gmtime, strftime\n",
    "\n",
    "reports_prefix = '{}/reports'.format(prefix)\n",
    "s3_report_path = 's3://{}/{}'.format(bucket,reports_prefix)\n",
    "\n",
    "mon_schedule_name = 'demo-xgboost-customer-churn-model-schedule-' + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "my_default_monitor.create_monitoring_schedule(monitor_schedule_name=mon_schedule_name,\n",
    "                                              endpoint_input=xgb_predictor.endpoint,\n",
    "                                              #record_preprocessor_script=pre_processor_script,\n",
    "                                              post_analytics_processor_script=s3_code_postprocessor_uri,\n",
    "                                              output_s3_uri=s3_report_path,\n",
    "                                              statistics=my_default_monitor.baseline_statistics(),\n",
    "                                              constraints=my_default_monitor.suggested_constraints(),\n",
    "                                              schedule_cron_expression=CronExpressionGenerator.hourly(),\n",
    "                                              enable_cloudwatch_metrics=True,\n",
    "                                             )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Start generating some artificial traffic\n",
    "The block below kicks off a thread to send some traffic to the created endpoint. This is so that we can continue to send traffic to the endpoint so that we will have always have data continually captured for analysis. If there is no traffic, the monitoring jobs will start to fail later on.\n",
    "\n",
    "Note that you need to stop the kernel to terminate this thread."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from threading import Thread\n",
    "from time import sleep\n",
    "import time\n",
    "\n",
    "runtime_client = boto3.client('runtime.sagemaker')\n",
    "\n",
    "# (just repeating code from above for convenience/ able to run this section independently)\n",
    "def invoke_endpoint(ep_name, file_name, runtime_client):\n",
    "    with open(file_name, 'r') as f:\n",
    "        for row in f:\n",
    "            payload = row.rstrip('\\n')\n",
    "            response = runtime_client.invoke_endpoint(EndpointName=ep_name,\n",
    "                                          ContentType='text/csv', \n",
    "                                          Body=payload)\n",
    "            time.sleep(1)\n",
    "            \n",
    "def invoke_endpoint_forever():\n",
    "    while True:\n",
    "        invoke_endpoint(endpoint_name, 'data/test-dataset-input-cols.csv', runtime_client)\n",
    "        \n",
    "thread = Thread(target = invoke_endpoint_forever)\n",
    "thread.start()\n",
    "\n",
    "# Note that you need to stop the kernel to stop the invocations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### List executions\n",
    "Once the schedule is scheduled, it will kick of jobs at specified intervals. Here we are listing the latest 5 executions. Note that if you are kicking this off after creating the hourly schedule, you might find the executions empty. You might have to wait till you cross the hour boundary (in UTC) to see executions kick off. The code below has the logic for waiting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mon_executions = my_default_monitor.list_executions()\n",
    "if len(mon_executions) == 0:\n",
    "    print(\"We created a hourly schedule above and it will kick off executions ON the hour.\\nWe will have to wait till we hit the hour...\")\n",
    "\n",
    "# UNCOMMENT the code below if you want to keep retrying until the hour\n",
    "\n",
    "# while len(mon_executions) == 0:\n",
    "#     print(\"Waiting for the 1st execution to happen...\")\n",
    "#     time.sleep(60)\n",
    "#     mon_executions = my_default_monitor.list_executions()  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate the latest execution and list the generated reports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "latest_execution = mon_executions[-1]\n",
    "print(\"Latest execution result: {}\".format(latest_execution.describe()['ExitMessage']))\n",
    "report_uri = latest_execution.output.destination\n",
    "\n",
    "print(\"Found Report Files:\")\n",
    "S3Downloader.list(report_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### List violations\n",
    "\n",
    "If there are any violations compared to the baseline, it will be generated here. Let's list the violations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "violations = my_default_monitor.latest_monitoring_constraint_violations()\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "constraints_df = pd.io.json.json_normalize(violations.body_dict[\"violations\"])\n",
    "constraints_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can plug in the processing job arn for a single execution of the monitoring into this notebook to see more detailed visualizations of the violations and distribution statistics of the data captue that was processed in that execution\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Clean-up\n",
    "\n",
    "If you're ready to be done with this notebook, please run the cell below.  This will remove the hosted endpoint you created and avoid any charges from a stray instance being left on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.delete_monitoring_schedule(mon_schedule_name)\n",
    "sess.delete_endpoint(xgb_predictor.endpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3 (Base Python)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-2:429704687514:environment/base-python"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
